Repository navigation
Change sync to cp - #169
Conversation
| fi | ||
|
|
||
| aws s3 sync "${LOCAL}/" "${REMOTE}/" --no-progress | ||
| aws s3 cp "${LOCAL}/" "${REMOTE}/" --recursive --no-progress |
There was a problem hiding this comment.
With sync, the idea is to only upload what has changed, so it lists every object in the relevant s3 dir, then makes the comparison. I suspect this is why the upload takes so long.
cp just copies the file, and overwrites it if there is an existing file, which is orders of magnitude faster.
--recursive docs: --recursive (boolean) Command is performed on all files or objects under the specified directory or prefix.
I don't think that we need what sync provides, but maybe I'm missing something!
jeancochrane
left a comment
jeancochrane
left a comment
There was a problem hiding this comment.
I'm down for this change! This thread and its subsequent replies contain some interesting discussion of cases in which sync is a better choice than cp, namely when there are lots of unchanged files in the source and you want to reduce your spend by avoiding pushing them to the destination. However, we tend to expect to overwrite all files when we do deploys these days, so I think the speed and simplicity of cp makes sense for our use case.
2 participants
Single pin with
sync: https://github.com/ccao-data/homeval/actions/runs/36770707995/job/110076187845Single pin with
cp: https://github.com/ccao-data/homeval/actions/runs/36772266760/job/110081453674Closes #91